Systems and methods for determining whether to modify content

ABSTRACT

The system determines whether content such as an image is suitable for content modification based on one or more criteria. The system includes decision engines or modules configured to evaluate one or more suitability metrics based on corresponding criteria such as publication status, restriction status, context, compatibility, and classification. If content is unsuitable for content modification because of entities or context depicted therein, privacy status, incompatibility with content modification, properties of the content file itself, or other aspects, the system generates a tag indicating the content is unsuitable for content modification. If content is suitable for content modification because of entities or context depicted therein, publication status, compatibility with content modification, properties of the content file itself, or other aspects, the system generates a content modification tag indicating the content is suitable for content modification. Metadata associated with the content is updated based on the flag.

BACKGROUND

The present disclosure is directed to evaluating images, and, moreparticularly, to systems for evaluating context and publishinginformation of an image to determine whether it is a candidate formodification.

SUMMARY

Photo-editing and processing software use object recognition to identifypeople or objects within photos. For example, applications offer usersthe ability to automatically tag their photos by auto-identifying facesto identify who is in the photos. In a further example, someapplications search for photos based on content based on a query orprompt (e.g., “Show me my images from Sicily”), with no need for taggingby the user. The ability to identify particular photo elements allowsfor their editing, adjustment, or even replacement. Should anapplication leverage such techniques for content modification, someproblems may arise. For example, photos might be selected formodification in a way that violates the privacy of a user, is offensiveto a user, is illegal, or is otherwise inappropriate. For example, auser may find editing of a private photo or a photo of a wedding to beinappropriate. Disclosed herein are systems and methods for screeningphotos or other content to determine whether content modification ispermissioned. For example, systems and methods disclosed herein mayavoid undesirable results by leveraging the fact users have alreadypublished a photo, and using further contextual analysis to identifysensitive or otherwise restricted photos.

Content is identified and analyzed to determine whether it is suitablefor modification. The analysis includes, for example, a determinationwhether the content is published or otherwise public, a determinationwhether the content is restricted, a determination whether the contentis compatible with modification, or a combination thereof. The analysisis based on processing of the content, metadata associated with thecontent, or both. The content is identified based on a request formodification, based on content-mining, or at the time of generation ofthe content, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the disclosure will beapparent upon consideration of the following detailed description, takenin conjunction with the accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which:

FIG. 1 is a block diagram of an illustrative content analyzer evaluatingtwo images, in accordance with some embodiments of the presentdisclosure;

FIG. 2 is a block diagram of an illustrative system for screening animage, in accordance with some embodiments of the present disclosure;

FIG. 3 is a block diagram of the illustrative image of FIG. 2 havingcorresponding metadata, in accordance with some embodiments of thepresent disclosure;

FIG. 4 is a block diagram of an illustrative system communicating over anetwork, in accordance with some embodiments of the present disclosure;

FIG. 5 is a block diagram of an illustrative system for locating andanalyzing content at one or more user devices over a network, inaccordance with some embodiments of the present disclosure;

FIG. 6 is a block diagram of an illustrative system for locating andanalyzing content over a network, in accordance with some embodiments ofthe present disclosure;

FIG. 7 shows an illustrative display of a user device with anapplication operating, in accordance with some embodiments of thepresent disclosure;

FIG. 8 is a block diagram of an illustrative user equipment, inaccordance with some embodiments of the present disclosure;

FIG. 9 is a flowchart of an illustrative process for analyzing contentfor content modification, in accordance with some embodiments of thepresent disclosure; and

FIG. 10 is a flowchart of an illustrative process for creating andmodifying content, in accordance with some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of an illustrative content analyzer evaluatingtwo images 101 and 102, in accordance with some embodiments of thepresent disclosure. Image 101, for example, includes an adult personwith a table in the background. Image 102, for example, includes oneadult person and two children. Content analyzer 110 screens each ofimages 101 and 102 to determine if they are suitable for modification(e.g., adding item 121). As illustrated, image analyzer 110 deems image101 to be suitable for modification. As illustrated, content analyzer110 deems image 102 to be unsuitable for modification, and accordinglyimage 102 is not modified. Content analyzer 110 may be implemented asany suitable combination of software and hardware, spanning one or moredevices.

Content analyzer 110 determines whether an image is suitable for contentmodification based on one or more criteria. In some embodiments, contentanalyzer 110 includes one or more decision engines or modules configuredto evaluate one or more suitability metrics based on correspondingcriteria. For example, illustrative content analyzer 110 includes orrelies on:

Publication Engine 151 Determines whether content is fully public,partially public, or private Restriction Engine 152 Determines whethercontent is restricted from modification Context Engine 153 Determinescontext information for the content Compatibility Engine 154 Determinescompatibility of the content with modification Classification Engine 155Determines a classification of the content Reference Database 156Includes reference information for screening content

In some embodiments, publication engine 151 determines whether contentis suitable for publication or public consumption. An image or othercontent that is private or otherwise not public may be deemed unsuitablefor content modification because the intended consumer audience is smallor nonexistent. Further, private content may not be intended for publicconsumption, and may include images, video, or audio that is sensitiveor private in nature. Publication engine 151 determines whether contentis public or is a candidate for being made public based on availableinformation (e.g., a metadata tag, a status identifier), determinedinformation (e.g., identification of a public instance of the content),or both.

In some embodiments, restriction engine 152 determines whether contentis restricted from modification. An image or other content that includesobjects, entities, context, or a combination thereof that matchrestriction criteria may cause the content to be restricted frommodification. For example, an image may include a child, a reveredentity, a religious site (e.g., a shrine, temple, or other structure),an event (e.g., a wedding, a funeral, or other event generallyconsidered off-limits to content modification), a questionable orsensitive object or act (e.g., a crime, a graphic image, or acontroversial image), any other attributes that may trigger restriction,or any combination thereof. In a further example, content may berestricted from modification based on preference information (e.g., auser has tagged a photo as restricted to modification), a classification(e.g., as determined by classification engine 155), any other suitableinformation, or any combination thereof. Restriction engine 152determines whether content is restricted from content modification basedon available information (e.g., a metadata tag, a status identifier),determined information (e.g., identification of a public instance of thecontent), or both.

In some embodiments, context engine 153 determines context informationfor content. Context information includes information describing what isdepicted or included in content such as a “who” (e.g., entities presentin the content), a “what” (e.g., an event or description), a “where”(e.g., a location, a building, a structure), a “when” (e.g., a time ofyear, a date, a time of day), any other suitable information, or acombination thereof. For example, context information for image 102 mayinclude a description such as:

Who child 1; child 2; Jane: mom What “family outdoors activity” Where“Concord, CA; ABC State Park” When “Mar. 1, 2019; 13:12:00”Context engine 153 determines context information of content based onavailable information (e.g., a metadata tag), determined information(e.g., identification and categorization of objects of the content), orboth.

In some embodiments, compatibility engine 154 determines whether contentis compatible with content modification, or for which types ofmodification the content is compatible with. For example, an image mayinclude a plurality of advertisements (e.g., a photo taken in TimesSquare in New York City), and compatibility engine 154 may determinethat the image is compatible with content modification. In a furtherexample, an image may already include a plurality of advertisements(e.g., the photo taken in Times Square in New York), and compatibilityengine 154 may determine that the image is compatible with certainproducts for content modification that do not conflict with the existingdepictions. In a further example, compatibility engine 154 may determinethat a quality, brightness, contrast, clarity, focus, arrangement, colorpalette, any other suitable visual aspect of an image, or anycombination thereof is not compatible with content modification.Compatibility engine 154 determines compatibility information based onavailable information (e.g., a metadata tag), determined information(e.g., identification and categorization of objects of the content,evaluation of the content), or both.

In some embodiments, classification engine 155 determines aclassification for the content based on a predetermined classificationscheme. For example, classification engine 155 may include a pluralityof classes of content, and may assign content to one or more classes. Insome embodiments, for example, the other engines and databases ofcontent analyzer 110 perform their respective processes based on aclassification assigned by classification engine 155. In someembodiments, for example, classification engine 155 classifies thecontent based on determinations or information of the other engines anddatabases of content analyzer 110. Classification engine 155 classifiesthe content based on available information (e.g., a metadata tag),determined information (e.g., identification and categorization ofobjects of the content, evaluation of the content), or both.

In some embodiments, reference database 156 stores reference informationthat may be retrieved by the engines of content analyzer 110. Referenceinformation includes, for example, reference classes or classifications,reference content, reference restriction categories, reference contextcategories, reference compatibility categories, reference contentsignatures, reference object signatures, any other suitable information,or any combination thereof.

In an illustrative example, content analyzer 110 identifies content suchas image 101 or image 102. Content analyzer 110 then determines whetherthe content is suitable for content modification based on at least oneof publication information (e.g., whether content is published and underwhat constraints), restriction information (e.g., whether contentcorresponds to a content modification blacklist), context information(e.g., whether the content's context is suitable for contentmodification), compatibility information (e.g., whether the content iscompatible with content modification), classification (e.g., whether thecontext's classification is suitable for content modification),reference information (e.g., used to compare with the content foridentification or other purposes), or any combination thereof.Considering image 101, content analyzer determines that image 101 issuitable for content modification. Considering image 102, contentanalyzer determines that image 102 is unsuitable for contentmodification. Content analyzer 110 need not modify an image deemedsuitable for modification. For example, modification may be performed bya separate application, device, application instance, or processingengine.

FIG. 2 is a block diagram of illustrative system 200 for screening animage, in accordance with some embodiments of the present disclosure.FIG. 3 is a block diagram of the illustrative image of FIG. 2 havingcorresponding metadata, in accordance with some embodiments of thepresent disclosure. Content analyzer 250 identifies information 210associated with an image under consideration. Content analyzer 250retrieves further information from database 220, in some circumstances,to aid in analyzing the image. Content analyzer 250 identifiesattributes 261-265 of the image and based on these attributes, and otherdeterminations, directs metadata manager 270 to update metadataassociated with the image. In some embodiments, for example, contentanalyzer 250, or components thereof, are used to perform contentmodification analysis. In some embodiments, for example, contentanalyzer 250 and metadata manager 270, or components thereof, are usedto perform content tagging (e.g., a tagging engine). In someembodiments, for example, content analyzer 250 and metadata manager 270,or components thereof, are used to identify and retrieve content (e.g.,a photo selection engine).

Information 210 includes content data (e.g., the image itself) andmetadata corresponding to the content. For example, as illustrated, theimage depicts three people, a vehicle, and a building, and the metadataincludes an image identifier (e.g., a file name), a time/date stampindicating a time of creation for the image, a location tag indicatingwhere the image was taken (e.g., generated when the content is created),metadata tags (e.g., keywords, scores, categorizations, and/or any othersuitable information), and image details (e.g., quality, resolution,user-defined descriptions, and/or any other suitable information). Theimage may be stored in any suitable file format, as a raster image or avector image. For example, the image may be stored as a JPEG, TIFF, GIF,BMP, PNG, SVG, or any other suitable file type. In a further example,the image may be stored using any suitable compression technique thatmay modify the file size of the image data. The metadata may be storedusing any suitable file type, having any suitable data structure andorganization. For example, the metadata may be stored as a collection offields or elements each having one or more entries (e.g., string arrays,numbers, or other data objects). In a further example, the metadata maybe stored in the same file as the content data, or may be stored in aseparate file (e.g., with the content data file including a reference tothe metadata file).

As illustrated in FIG. 3, information 210 includes image 211 andmetadata 201-205 that correspond to image 211. Illustrative metadata201-205 may be rearranged, added to, reduced, or otherwise modified, orotherwise assume any suitable format, in accordance with the presentdisclosure. The illustrative partitioning of metadata illustrated inFIG. 3 need not be followed. For example, the metadata may be organizedas a single grouping of information with suitable organization and/orcategorization.

Identifier 201, as illustrated, includes a device ID (e.g., identifyinga device from which the content is generated or stored), a file name(e.g., of the content data, the metadata, or both), and a reference ID(e.g., used as an auxiliary identifier for the image, and may beuser-defined). In some embodiments, identifier 201 includes a devicehardware address, a device IP address, a device nickname (e.g., asdefined by a DNS-type server), any other suitable identificationinformation, or any combination thereof. In some embodiments, identifier201 includes a file name, a file extension, a file type, a file size,directory information for where the file is stored in memory, any othersuitable file-related information, or any combination thereof. In someembodiments, identifier 201 includes a file index, a file nickname(e.g., as specified by a web-based application), a file name ofaccompanying metadata, a file location of accompanying metadata, anyother suitable reference information, or any combination thereof.

Time/date stamp 202, as illustrated, includes a time of generation(e.g., when the image was created), a time when content was saved (e.g.,when the image was first or most recently saved), a time content wasuploaded (e.g., when the image was uploaded to a web-based application),a time when content was retrieved (e.g., when the image was retrieved bycontent analyzer 250), and a time when content was modified (e.g., whenthe image was modified, if at all). In some embodiments, time/date stamp202 includes a temporal history of transactions and events involvingimage 211. For example, a time of creation for each instance of image211, as saved in memory of one or more devices, may be included intime/date stamp 202. In some embodiments, time/date stamp 202 may beapplication-specific. For example, a social media application may storean instance of information 210, with time/date stamp 202 includinginformation relative to image 211 being stored and identified by theapplication (e.g., the creation date is the date the image is submittedto the application).

Location 203, as illustrated, includes, a user-defined location (e.g.,as input by the user to a touchscreen), GPS data (e.g., coordinates orlocations based on GPS data), map data, and an entity identifier (e.g.,a site name such as “Citi Field”). In some embodiments, location 203includes a keyword tag inputted by a user (e.g., “Phoenix,” “Mexico,”“Yosemite National Park”). In some embodiments, location 203 includes aGPS tag including raw data, from each of a plurality of satellites, suchas a satellite clock value, ephemeris values, and almanac values. Insome embodiments, location 203 includes a GPS tag including time,longitude, latitude, altitude, and number of satellites the data isbased on. In some embodiments, location 203 includes a two-dimensionalreference coordinate pair in the context of a digital map. For example,location 203 may include area designation A5 of an alphanumericallypartitioned map region, with letters denoting one dimension and numbersdenoting the orthogonal dimension. In some embodiments, location 203includes a device-generated tag based on GPS data. For example, location203 may include a street name, intersection identifier, neighborhoodname, city name, state name, country name, park name, body of watername, geological feature name (e.g., Grand Canyon, Mojave Desert),geological feature description (e.g., mountain, forest, creekside), anyother suitable tag corresponding to a location, or any combinationthereof.

Tags 204, as illustrated, include attributes, entities, privacysettings, restriction settings, descriptions, user preferences,preferences, compatibilities, and keywords. Attributes may include, forexample, people, places, events, items, arrangements thereof, any othersuitable aspect depicted in content, or any combination thereof. Forexample, an attribute may include a particular person (e.g., Jon Doe), aperson identifier (e.g., adult, woman, boy, baby), a binary indicationof whether any person is depicted in the content, any other suitableperson-related information, or any combination thereof. In a furtherexample, attributes may include an event name (e.g., “Christmas,”“Boston Marathon,” “Trip to NYC”), an item identifier (e.g., a car, aboat, a building, a beverage, a computer, a user device, a television),an arrangement of items (e.g., a set dinner table, a group of people, acar race, a cityscape), any other suitable descriptors for aspectsdepicted in content, or any combination thereof.

Details 205, as illustrated, include owner input, file history,modifications, posted comments, file properties, and image quality. Insome embodiments, details 205 include input from a user or content owner(e.g., descriptions, preferences, or status identifiers), a filetransaction history (e.g., receiving, transmitting, storing, recalling,modifying), content modifications (e.g., a mapping of changed pixels ina rasterized image), a history of users' interaction with image 211(e.g., comments, views, user tags such as “likes”), properties of thecontent data file or metadata file (e.g., file size, file location, filecompression), a quality metric of image 211 (e.g., a resolution in unitsof pixels, a color scale such as color/grayscale/black-and-white,quality identifier such as good, bad, or blurry), any other suitableinformation, or any combination thereof.

Content analyzer 250, similar to content analyzer 150 of FIG. 1,includes publication identifier 251, restriction identifier 252, contentidentifier 253, compatibility identifier 254, image classifier 255, anddatabase 256 (also each referred to as “application modules”). Contentanalyzer 250 may include any or all of the functionality of applicationmodules 251-256, which are merely illustrative and need not be separatemodules (e.g., a single module may include all of their functionality).In some embodiments, content analyzer 250 is capable of retrievinginformation from external database 220. By applying any or all ofapplication modules 251-256, content analyzer 250 determines whetherimage 211 is suitable for content modification, what type of contentmodification is suitable or unsuitable, or a combination thereof. Insome embodiments, the more particular determination and identificationof objects depicted in content is desired for content modificationpurposes. For example, determination of advertisement audience andcontent modification appropriateness may aid in the results of contentmodification (e.g., recognition and reaction by consumers), andaccordingly further detailed information of image 211 may be useful. Asillustrated, content analyzer 250 has identified attributes includinglocation 261 (e.g., “A. Center”), person 262 (e.g., adult 1), person 263(e.g., adult 2), person 264 (e.g., child 1), and vehicle 265 (e.g.,“2016 Fiat 500X”). Content analyzer 250 need not identify all attributesof a content item, and may identify any suitable number of attributes(e.g., five as illustrated in FIGS. 2-3). Context identifier 253 may beconfigured to identify objects of image 211 using any suitable techniquesuch as pattern recognition techniques, artificial intelligence-basedmachine learning techniques, any other suitable techniques, or anycombination thereof.

In some embodiments, publication identifier 251 determines whether image211 is public, publicly available, private, shared, or undesignated. Insome embodiments, publication identifier 251 accesses metadata (e.g.,tags 204, details 205, or identifier 201) to determine whether image 211has been previously tagged as public or private (e.g., by a user, anapplication, or a device). In some embodiments, publication identifier251 determines whether image 211 is public based on whether image 211 ispublicly available (e.g., displayed in a social media application feed,displayed on a webpage). For example, publication identifier 251 maygenerate one or more digital signatures corresponding to the image,describing features and aspects of the image. Publication identifier 251may then compare the digital signatures against one or more referencesignatures of publicly available content to identify the image as beingpublicly available. In some circumstances, publication identifier 251may have insufficient information to make a determination of whether thecontent is public. In some such circumstances, publication identifier251 may determine whether the content is suitable for publication. Insome circumstances, publication identifier 251 may determine content isprivate and not suitable for content modification at all, or is suitablefor limited content modification.

In some embodiments, restriction identifier 252 determines whethercontent is restricted from content modification, publication, orotherwise any modification. In some embodiments, restriction identifier252 accesses metadata (e.g., tags 204, details 205, or identifier 201)to determine whether image 211 is restricted from modification. Forexample, the tags 204 may include a restriction status identifier suchas “restricted’ or “unrestricted.” In a further example, restrictionidentifier 252 may access tags 204 to determine that image 211 includesa child (e.g., person 264), a spiritual place (e.g., location 261), orboth, and in response, determines that image 211 is restricted fromcontent modification.

In some embodiments, context identifier 253 generates digital signaturescorresponding to the identified objects, describing features and aspectsof the objects. In some such embodiments, context identifier 253 isconfigured to compare the digital signatures against one or morereference signatures to identify the objects based on known aspects ofthe reference signatures. For example, context identifier 253 mayidentify person 262 as a person depicted in image 211, and then maycompare digital signatures of person 262 against known digitalsignatures for one or more identified users (e.g., Jon Doe) to identifyperson 262 as Jon Doe. Accordingly, context identifier 253 may identifyattributes of image 211 generally (e.g., a person, a child) or moreparticularly (e.g., by the person's name, gender, age range, race, orother descriptor). In a further example, context identifier 253 mayidentify a group of people (e.g., including person 262, 263, and 264),and that one person of the group is a child (e.g., person 264). In afurther example, context identifier 253 may identify vehicle 265generally, and then compare one or more digital signatures for vehicle265 against reference digital signatures of known vehicles to identifyvehicle 265. In a further example, context identifier 253 may identify alocation “A. Center” corresponding to a business or non-profit entity.In an illustrative example, “A. Center” may be a meditation center, andcontext identifier 253 may identify a context as spiritual or religioussetting, which may impact whether content modification is appropriate.In a further illustrative example, “A. Center” may be a place ofbusiness such as a music store, and context identifier 253 may identifya context as commercial setting, which may impact whether contentmodification is appropriate. In some embodiments, context identifier 253identifies text, numbers, words, or other symbols in image 211, andextracts keywords (e.g., to generate a search query) to furtherdetermine attributes depicted in image 211. For example, contextidentifier 253 may identify the text “A. Center” and along with a GPStag, determine that the object is a commercial entity “A. Center, LLC”having an address, business type, open hours, social media applicationprofile, and product type.

In some embodiments, compatibility identifier 254 determines whetherimage 211 is compatible with content modification or for what type ofmodification image 211 is suitable. For example, an image may includeonly natural scenes (e.g., a photo of a forest taken during hiking) withno suitable objects or areas that lend themselves to contentmodification. In some such circumstances, compatibility identifier 254may determine that the image is not compatible with contentmodification. In some embodiments, compatibility identifier 254determines whether content is compatible with content modification basedon the metadata. For example, tags 204 may include tags indicating thatimage 211 depicts a private family outing, and compatibility identifier254 may determine that image 211 is not compatible with contentmodification. In some embodiments, compatibility identifier 254 accessesmetadata to determine what objects are depicted in the content.

In some embodiments, image classifier 255 classifies content based onany of the application modules of content analyzer 250, based onmetadata, or both. In some embodiments, image classifier determines acategory for an image based on which type of modification is suitable ornot suitable for the image. For example, image classifier 255 mayclassify images depicting street scenes (e.g., having people, buildings,cars, and generally many logos) as suitable for any type of contentmodification. In a further example, for images depicting less objects,image classifier 255 may classify the image as being suitable for onlycertain types of products, if any.

In some embodiments, databases 220 and 256 store information and contentthat may be accessed by content analyzer 250. As illustrated, database256 is local to content analyzer 250 (e.g., implemented on the samecomputing hardware or device). For example, database 256 may be storedin memory of a user device on which content analyzer 250 is executed. Asillustrated, database 220 is external to content analyzer 250, andaccordingly may be remote from content analyzer 250.

In some embodiments, the determination by content analyzer 250 regardingcontent modification need not be performed explicitly at the time ofanalysis. For example, content analyzer 250 may generate a contentmodification flag that is indicative of whether, and what type of,content modification is appropriate. The flag may be evaluated at alater time, for example, when a request to modify an image for contentmodification is received. Accordingly, content analyzer 250 may analyzecontent at any suitable point between, and including, content generationand reception of a request for content modification. For example,content analyzer 250 may analyze content as it is created or may analyzecontent at any suitable point in time after creation including inresponse to an explicit request. In an illustrative example, in theabsence of a request for content modification, content analyzer 250 maymine one or more network entities or devices for stored content, analyzethe content for content modification suitability, and generate acorresponding content modification flag for storage (e.g., for use whena request is received). In some embodiments, content analyzer 250generates one or more tags that may be used to evaluate, for a givenmodification type, if an image is suitable.

Metadata manager 270, which may be, but need not be, included as part ofcontent analyzer 250 (e.g., as illustrated, they are separateapplications). Metadata manager 270 updates metadata of information 210,adds tags to metadata of information 210, generates metadata ofinformation 210, or otherwise manages updating and storing of metadatacorresponding to image 211. For example, in some embodiments, contentanalyzer 250 generates a products placement flag and transmits the flagto metadata manager 270, which updates the existing metadata ofinformation 210 based on the flag. In some embodiments, metadata manager270 reads and writes files to memory (e.g., local memory storage orremote storage via a communications network). For example, in someembodiments, metadata manager 270 opens a content data file and metadataand provides that information to content analyzer 250. In a furtherexample, content analyzer 250 may generate tags or other changes tometadata, which may be provided to metadata manager 270, which in turnwrites this information to an appropriate file and saves the files insuitable storage.

FIGS. 4-6 illustrate systems having network-based arrangements. Forexample, because content is typically generated and stored among userdevices and network devices, and transmitted via communicationsnetworks, aspects of the present disclosure will be discussed in thecontext of the illustrative systems of FIGS. 4-6, although any suitablenetwork-based arrangement may be used, in accordance with the presentdisclosure.

FIG. 4 is a block diagram of illustrative system 400 communicating overa network, in accordance with some embodiments of the presentdisclosure. Illustrative system 400 may be representative ofcircumstances in which a user creates content at user device 450, viewscontent on a display of user device 450, downloads content from networkentity 420 to user device 450, uploads content from user device 450 tonetwork entity 420, or a combination thereof. In system 400, there maybe more than one of each type of user device, but only one of each isshown in FIG. 4 to avoid overcomplicating the drawing. In addition, eachuser may utilize more than one type of user device and also more thanone of each type of user device.

User device 450, illustrated as a wireless-enabled device, may becoupled to communications network 410 (e.g., connected to the Internet).For example, user device 450 is coupled to communications network 410via a communications path to access point 440. In some embodiments, userdevice 450 may be a computing device coupled to communications network410 via a wired connection. For example, user device 450 may alsoinclude wired connections to a LAN, or any other suitable communicationslink to network 410. Communications network 410 may be one or morenetworks including the Internet, a mobile phone network, mobile voice ordata network (e.g., a 4G or LTE network), cable network, public switchedtelephone network, or other types of communications network orcombinations of communications networks. Communications paths mayinclude one or more communications paths, such as a satellite path, afiber-optic path, a cable path, a path that supports Internetcommunications, free-space connections (e.g., for broadcast or otherwireless signals), or any other suitable wired or wirelesscommunications path or combination of such paths. Althoughcommunications paths are not drawn between user device 450 and networkentity 420, these devices may communicate directly with each other viacommunications paths, such as those described above, as well as othershort-range point-to-point communications paths, such as USB cables,IEEE 1394 cables, wireless paths (e.g., Bluetooth, infrared, IEEE802-11x, etc.), or other short-range communication via wired or wirelesspaths. BLUETOOTH is a certification mark owned by Bluetooth SIG, INC.Devices may also communicate with each other directly through anindirect path via communications network 410.

System 400 includes network entity 420 (e.g., a server or other suitablecomputing device) coupled to communications network 410 via a suitablecommunications path. Communications between network entity 420 and userdevice 450 may be exchanged over one or more communications paths butare shown as a single path in FIG. 4 to avoid overcomplicating thedrawing. Network entity 420 may include a database, one or moreapplications (e.g., as an application server, host server). A pluralityof network entities may exist and be in communication with network 410,but only one is shown in FIG. 4 to avoid overcomplicating the drawing.In some embodiments, network entity 420 may include one source device.In some embodiments, network entity 420 implements an application thatcommunicates with instances of applications at many user devices (e.g.,user device 450). For example, an instance of a social media applicationmay be implemented on user device 450, with application informationbeing communication to and from network entity 420 which may storeprofile information for user (e.g., so that a current social media feedavailable on other devices than user device 450).

In some embodiments, network entity 420 includes one or more types ofstored information, including, for example, publication information,restriction criteria, metadata, reference digital signatures, historicalcommunications records, user preferences, user profile information, anyother suitable information, or any combination thereof. Network entity420 may include an applications-hosting database or server, plug-ins, asoftware developers kit (SDK), an applications programming interface(API), or other software tools configured to provide software (e.g., asdownload to a user device), run software remotely (e.g., hostingapplications accessed by user devices), or otherwise provideapplications support to applications of user device 450. In someembodiments, information from network entity 420, is provided to userdevice 450 using a client-server approach. For example, user device 450may pull information from a server, or a server may push information touser device 450. In some embodiments, an application client residing onuser device 450 may initiate sessions with network entity 420 to obtaininformation when needed (e.g., when data is out-of-date or when a userdevice receives a request from the user to receive data). In someembodiments, information may include user information (e.g., userprofile information, user-created content). For example, the userinformation may include current and/or historical user activityinformation such as what content transactions the user engages in,whether the user interacts with a social network, at what times the userinteracts with a social network to post content, what types of contentthe user typically uploads, stored contacts of the user, frequentcontacts of the user, any other suitable information, or any combinationthereof. In some embodiments, the user information may identify patternsof a given user for a period of time.

In some embodiments, an application may be implemented on user device450, network entity 420, or both. For example, the application may beimplemented as software or a set of executable instructions which may bestored in storage of the user device 450, network entity 420, or both,and executed by control circuitry of the respective devices. In someembodiments, an application may include an imaging application, a photomanagement application, a content modification application, or acombination thereof, that is implemented as a client-server-basedapplication where only a client application resides on user device 450,and a server application resides on a remote server (e.g., networkentity 420). For example, an application may be implemented partially asa client application on user device 450 (e.g., by control circuitry ofuser device 450) and partially on a remote server as a serverapplication running on control circuitry of the remote server (e.g.,control circuitry of network entity 420). When executed by controlcircuitry of the remote server, the application may instruct the controlcircuitry to generate a display and transmit the generated display touser device 450. The server application may instruct the controlcircuitry of the remote device to transmit data for storage on userdevice 450. The client application may instruct control circuitry of thereceiving user device to generate the application displays.

In some embodiments, the arrangement of system 400 is a cloud-basedarrangement. The cloud provides access to services, such as informationstorage, messaging, or social networking services, among other examples,as well as access to any content described above, for user devices.Services can be provided in the cloud through cloud computing serviceproviders, or through other providers of online services. For example,the cloud-based services can include a storage service, a sharing site,a social networking site, or other services via which user-sourcedcontent is distributed for viewing by others on connected devices. Thesecloud-based services may allow a user device to store information to thecloud and to receive information from the cloud rather than storinginformation locally and accessing locally stored information. Cloudresources may be accessed by a user device using, for example, a webbrowser, a messaging application, a social media application, a desktopapplication, a mobile application, and/or any combination of accessapplications of the same. User device 450 may be a cloud client thatrelies on cloud computing for application delivery, or user device 450may have some functionality without access to cloud resources. Forexample, some applications running on user device 450 may be cloudapplications (e.g., applications delivered as a service over theInternet), while other applications may be stored and run on user device450. In some embodiments, user device 450 may receive information frommultiple cloud resources simultaneously.

In an illustrative example, a user may create images 480 and 470 using acamera integrated into user device 450. In some embodiments, the usermay use imaging application 460 to create images 470 and 480. Imagingapplication 460 may capable of file management, file tagging, filestoring, content modification, any other suitable functionality, or anycombination thereof. Imaging application 460 may be implemented as asoftware application on user device 450, or may be included as anapplication operating within a host program running on user device 450.In some embodiments, imaging application 460, or other suitableapplication, generates metadata 471 and 481 for respective images 470and 480 at the time of creation. For example, when a photo is taken, theapplication may query the user for information such as filename, filetype, tags, description, public/private status, any other suitableinformation, or any combination thereof. In some embodiments, imagingapplication 460 manages formatting and storing images 470 and 480, whilemanagement of metadata and tagging is performed by a separateapplication (e.g., at a later time than image creation). In someembodiments, images 470 and 480, among a plurality of images, are storedin memory of user device 450.

In an illustrative example, a user may download images from networkentity 420 via communications network 410 for viewing and storage inmemory of user device 450. For example, the user may download imagesfrom a social networking feed, a website, a file site, another device(e.g., via a USB connection), any other suitable repository, or anycombination thereof. Downloaded content may include metadata as part ofthe content file(s), separate metadata files, or both. In someembodiments, imaging application 460 generates metadata for downloadedimages.

In an illustrative example, a user may upload images to network entity420 via communications network 410 for storage in memory of networkentity 420. For example, the user may upload images to a socialnetworking feed, a website, a file site, another device (e.g., via a USBconnection), any other suitable repository, or any combination thereof.Uploaded content may include metadata as part of the content file(s),separate metadata files, or both. In some embodiments, imagingapplication 460 generates metadata for uploaded images prior touploading.

In an illustrative example, network entity 420 may search for contentamong devices coupled to communications network 410. For example,network entity 420 may communicate with user device 450 to discover newimages that may be stored on network entity 420. Further description ofsome such embodiments is provided in the context of FIG. 5, for example.

In an illustrative example, a photo selection engine may operate oncontrol circuitry of network entity 420, user device 450, or both,identifying and retrieving content from one or more devices (e.g., userdevice 450). User device 450, network entity 420, or both, may host atagging engine, which generates tags as content is created (e.g., byupdating metadata 481 and 471).

FIG. 5 is a block diagram of illustrative system 500 for locating andanalyzing content at one or more user devices 550-553 over network 510,in accordance with some embodiments of the present disclosure.Illustrative system 500 may be representative of circumstances in whichmultiple users create content at user devices 550, 551, 552, and 553 andview content on respective displays of user devices 550-553, downloadcontent from network entity 520 to user devices 550-553, upload contentfrom user devices 550-553 to network entity 520, or a combinationthereof. Each user may utilize more than one type of user device andalso more than one of each type of user device. Any of user devices550-553 may be the same as, or similar to, user device 450 of FIG. 4.

User devices 550-553 are coupled to communications network 510, whichmay be the same as, or different from, network 410 of FIG. 4. In someembodiments, each of user devices 550-553 may be a computing devicecoupled to communications network 510 via a wired connection, a wirelessconnection, any other suitable communications link to network 510, orany combination thereof. Communications network 510 may be one or morenetworks including the Internet, a mobile phone network, mobile voice ordata network, cable network, public switched telephone network, or othertypes of communications network or combinations of communicationsnetworks. Communications paths may include one or more communicationspaths, such as a satellite path, a fiber-optic path, a cable path, apath that supports Internet communications, free-space connections(e.g., for broadcast or other wireless signals), or any other suitablewired or wireless communications path or combination of such paths.Although direct communications paths are not drawn between user devices550-553 and network entity 520, these devices may communicate directlywith each other via communications paths, such as those described above,as well as other short-range point-to-point communications paths, suchas USB cables, IEEE 1394 cables, wireless paths (e.g., Bluetooth,infrared, IEEE 802-11x, etc.), or other short-range communication viawired or wireless paths. BLUETOOTH is a certification mark owned byBluetooth SIG, INC. Devices may also communicate with each otherdirectly through an indirect path via communications network 510.

System 500 includes network entity 520 (e.g., a server or other suitablecomputing device) coupled to communications network 510 via a suitablecommunications path. Communications between network entity 520 and userdevices 550-553 may be exchanged over one or more communications pathsbut are shown as a single path in FIG. 5 to avoid overcomplicating thedrawing. Network entity 520 may include a database, one or moreapplications (e.g., as an application server, host server), or both. Aplurality of network entities may exist and be in communication withnetwork 510, but only one is shown in FIG. 5 to avoid overcomplicatingthe drawing. In some embodiments, network entity 520 may include onesource device. In some embodiments, network entity 520 implements anapplication that communicates with instances of applications at manyuser devices (e.g., user devices 550-553). For example, an instance of asocial media application may be implemented on each of user devices550-553, with application information being communication to and fromnetwork entity 520 which may store profile information for each user(e.g., so that a current social media feed available on other devicesthan user devices 550-553). In some embodiments, network entity 520includes one or more types of stored information, including, forexample, publication information, restriction criteria, metadata,reference digital signatures, historical communications records, userpreferences, user profile information, any other suitable information,or any combination thereof. Network entity 520 may include anapplications-hosting database or server, plug-ins, a software developerskit (SDK), an applications programming interface (API), or othersoftware tools configured to provide software (e.g., as download to auser device), run software remotely (e.g., hosting applications accessedby user devices), or otherwise provide applications support toapplications of user devices 550-553. In some embodiments, informationfrom network entity 520, is provided to user devices 550-553 using aclient-server approach. For example, user devices 550-553 may each pullinformation from a server, or a server may push information to each ofuser devices 550-553. In some embodiments, an application clientresiding on any of user devices 550-553 may initiate sessions withnetwork entity 520 to obtain information when needed (e.g., when data isout-of-date or when a user device receives a request from the user toreceive data). In some embodiments, information may include userinformation (e.g., user profile information, user-created content). Forexample, the user information may include current and/or historical useractivity information such as what content transactions the user engagesin, whether the user interacts with a social network, at what times theuser interacts with a social network to post content, what types ofcontent the user typically uploads, stored contacts of the user,frequent contacts of the user, any other suitable information, or anycombination thereof. In some embodiments, the user information mayidentify patterns of a given user for a period of time.

In some embodiments, an application may be implemented on user devices550-553, network entity 520, or a combination thereof. For example, theapplication may be implemented as software or a set of executableinstructions which may be stored in storage of each of user devices550-553, network entity 520, or a combination thereof, and executed bycontrol circuitry of the respective devices. In some embodiments, anapplication may include an imaging application, a photo managementapplication, a content modification application, or a combinationthereof, that is implemented as a client-server-based application whereonly a client application resides on user devices 550-553, and a serverapplication resides on a remote server (e.g., network entity 520). Forexample, an application may be implemented partially as a clientapplication on user devices 550-553 (e.g., by control circuitry ofrespective user devices 550-553) and partially on a remote server as aserver application running on control circuitry of the remote server(e.g., control circuitry of network entity 520). When executed bycontrol circuitry of the remote server, the application may instruct thecontrol circuitry to generate respective displays and transmit thegenerated displays to user devices 550-553. The server application mayinstruct the control circuitry of the remote device to transmit data forstorage on any or all of user devices 550-553. The client applicationmay instruct control circuitry of the receiving user device to generatethe application displays.

In some embodiments, the arrangement of system 500 is a cloud-basedarrangement. The cloud provides access to services, such as informationstorage, messaging, or social networking services, among other examples,as well as access to any content described above, for user devices.Services can be provided in the cloud through cloud computing serviceproviders, or through other providers of online services. For example,the cloud-based services can include a storage service, a sharing site,a social networking site, or other services via which user-sourcedcontent is distributed for viewing by others on connected devices. Thesecloud-based services may allow a user device to store information to thecloud and to receive information from the cloud rather than storinginformation locally and accessing locally stored information. Cloudresources may be accessed by a user device using, for example, a webbrowser, a messaging application, a social media application, a desktopapplication, a mobile application, and/or any combination of accessapplications of the same. User devices 550-553 may be cloud clients thatrely on cloud computing for application delivery, or user devices550-553 may have some functionality without access to cloud resources.For example, some applications running on user devices 550-553 may becloud applications (e.g., applications delivered as a service over theInternet), while other applications may be stored and run on userdevices 550-553. In some embodiments, user devices 550-553 may receiveinformation from multiple cloud resources simultaneously.

In an illustrative example, a user may create image 570 using a cameraintegrated into user device 551. In some embodiments, the user may usean application to create, edit, format, and store image 570. Theapplication may be implemented as a software application on user device551, or may be included as an application operating within a hostprogram running on user device 551. In some embodiments, the applicationgenerates metadata 571 for image 570 at the time of creation. Forexample, when a photo is taken, the application may query the user forinformation such as filename, file type, tags, description,public/private status, any other suitable information, or anycombination thereof. Images 570, among a plurality of images, is storedin memory of user device 551.

In an illustrative example, the user associated with user device 551 mayupload, or otherwise share image 570 with other users by making image570 publicly available. An application on user device 551 may manageuploading image 570 and metadata 571 to network entity 520, which may inturn update one or more instances of the application on user devices550, 552, and 553 to reflect the publication of image 570. In someembodiments, the application may update metadata 571 to reflect thepublication of image 570.

In an illustrative example, network entity 520 searches among devicescoupled to network 510 for content that may be considered for contentmodification. In some embodiments, network entity 520 may host anapplication to user devices 550-553. The application may determine whatcontent is stored in each of user devices 550-553, and for any contentnot already considered for content modification, the application maydetermine whether the content is suitable for content modification(e.g., as described in the context of FIGS. 1-3). In some embodiments,network entity 520 generates metadata, updates existing metadata, orboth in response to analyzing content for suitability for contentmodification. For example, network entity 520 may identify one or moreproducts for placement, and then search among content stored on, orassociated with users of (e.g., and stored on network entity 520), userdevices 550-553.

In an illustrative example, applications operating on user devices550-553 may communicate with one another (e.g., via a WiFi, or datanetwork such as 4G). In some embodiments, a user may create and storeimage 570 locally. If the user does not upload (e.g., to a cloud-basedstorage device), post (e.g., on a social media feed), share (e.g., via amessaging application), or otherwise make image 570 public, other userdevices (e.g., user devices 550, 552, and 553) and network entities(e.g., network entity 520) may have limited access to image 570.Further, even with access to unpublished image 570, an applicationsearching for content for which content modification is suitable mayrecognize that image 570 is not published and determine that it is notsuitable for content modification. The application may determine thatimage 570 is not published based on metadata 571, lack of uploading,posting, or sharing of image 570, or both. If the user does upload,post, share, or otherwise make image 570 public, then an applicationoperating on user devices 550, 552, and 553, and network entity 520 maydetermine image 570 is published, and therefore suitable to beconsidered for content modification (e.g., subject to analysis based onother criteria as illustrated in FIGS. 1-3).

In an illustrative example, a photo selection engine may operate oncontrol circuitry of any or all of network entity 520 and user devices550-553, identifying and retrieving content from one or more devices(e.g., user devices 550-553). User devices 550-553, network entity 520,or a combination thereof, may host a tagging engine, which generatestags as content is created (e.g., by updating metadata 571). A contentmodification engine may operate on control circuitry of network entity520, user devices 550-553, or a combination thereof, analyzing contentfor content modification suitability.

FIG. 6 is a block diagram of illustrative system 600 for locating andanalyzing content over network 610, in accordance with some embodimentsof the present disclosure. Illustrative system 600 may be representativeof circumstances in which multiple network entities (e.g., networkentities 620, 21, 622, 623, and 624) communicate with each other overnetwork 610. Network entities 620-624 may include user devices, servers,application servers, cloud-based databases, computers (e.g., personalcomputers, laptops, tablets), any other suitable computing equipment, orany combination thereof.

Network entities 620-624 are coupled to communications network 610,which may be the same as, or different from, network 410 of FIG. 4 ornetwork 510 of FIG. 5. In some embodiments, network entities 620-624 maybe computing devices coupled to communications network 610 via a wiredconnection, a wireless connection, any other suitable communicationslink to network 610, or any combination thereof. Communications network610 may be one or more networks including the Internet, a mobile phonenetwork, mobile voice or data network, cable network, public switchedtelephone network, or other types of communications network orcombinations of communications networks. Communications paths mayinclude one or more communications paths, such as a satellite path, afiber-optic path, a cable path, a path that supports Internetcommunications, free-space connections (e.g., for broadcast or otherwireless signals), or any other suitable wired or wirelesscommunications path or combination of such paths. Although directcommunications paths are not drawn between network entities 620-624,these devices may communicate directly with each other viacommunications paths, such as those described above, as well as othershort-range point-to-point communications paths, such as USB cables,IEEE 1394 cables, wireless paths (e.g., Bluetooth, infrared, IEEE802-11x, etc.), or other short-range communication via wired or wirelesspaths. BLUETOOTH is a certification mark owned by Bluetooth SIG, INC.Devices may also communicate with each other directly through anindirect path via communications network 610.

System 600 includes network entities 620-624 (e.g., each being a serveror other suitable computing device) coupled to communications network610 via a suitable communications path. Communications between networkentities 620-624 may be exchanged over one or more communications pathsbut are shown as a single path in FIG. 6 to avoid overcomplicating thedrawing. Network entities 620-624 may each include a database, anapplication server, a host server), or a combination thereof. In someembodiments, network entities 620-624 implement an application thatcommunicates with instances of applications at other network entities.For example, an instance of an imaging application may be implemented oneach of network entities 620-624, with application information beingcommunication to and from the network entities, which may store profileinformation for a plurality of users. In some embodiments, networkentities 620-624 each include one or more types of stored information,including, for example, publication information, restriction criteria,metadata, reference digital signatures, historical communicationsrecords, user preferences, user profile information, any other suitableinformation, or any combination thereof. Any or all of network entities620-624 may include an applications-hosting database or server,plug-ins, a software developers kit (SDK), an applications programminginterface (API), or other software tools configured to provide software(e.g., as download to a user device), run software remotely (e.g.,hosting applications accessed by user devices), or otherwise provideapplications support to applications of network entities 620-624. Insome embodiments, information from any of network entities 620-624 isprovided to other of network entities 620-624 using a client-serverapproach. For example, network entities 620-624 may each pullinformation from a server, o reach other, or a server may pushinformation to any or all of network entities 620-624. In someembodiments, an application client residing on any of network entities620-624 may initiate sessions with any other network entity to obtaininformation when needed (e.g., when data is out-of-date or when a userdevice receives a request from the user to receive data). In someembodiments, information may include content, metadata, userinformation, content modification criteria information, any othersuitable information, or any combination thereof.

In some embodiments, an application may be implemented on networkentities 620-624. For example, the application may be implemented assoftware or a set of executable instructions which may be stored instorage of each of network entities 620-624, and executed by controlcircuitry of the respective devices. In some embodiments, an applicationmay include an imaging application (e.g., for creating, editing, andfiling content in storage), a photo management application (e.g., forstoring and/or modifying content), a content modification application(e.g., for analyzing content for suitability of content modification), acontent identifying application (e.g., for identifying content amongnetwork devices), or a combination thereof, that is implemented as aclient-server-based application where only a client application resideson some of network entities 620-624, and a server application resides onother of network entities 620-624. For example, an application may beimplemented partially as a client application on network entities620-624 (e.g., by control circuitry of respective network entities620-624) and partially on a remote server as a server applicationrunning on control circuitry of the remote server.

In some embodiments, the arrangement of system 600 is a cloud-basedarrangement. The cloud provides access to services, such as informationstorage, messaging, or social networking services, among other examples,as well as access to any content described above, for user devices.Services can be provided in the cloud through cloud computing serviceproviders, or through other providers of online services. For example,the cloud-based services can include a storage service, a sharing site,a social networking site, or other services via which user-sourcedcontent is distributed for viewing by others on connected devices. Thesecloud-based services may allow a network entity to store information tothe cloud and to receive information from the cloud rather than storinginformation locally and accessing locally stored information. Cloudresources may be accessed by a network entity or user device using, forexample, a web browser, a messaging application, a social mediaapplication, a desktop application, a mobile application, and/or anycombination of access applications of the same. Network entities 620-624may be cloud clients that rely on cloud computing for applicationdelivery, or network entities 620-624 may have some functionalitywithout access to cloud resources. For example, some applicationsrunning on network entities 620-624 may be cloud applications (e.g.,applications delivered as a service over the Internet), while otherapplications may be stored and run on network entities 620-624. In someembodiments, network entities 620-624 may receive information frommultiple cloud resources simultaneously.

In an illustrative example, network entity 620 may identify image 670having associated metadata 671. Network entity 620 may analyze image 670for suitability for content modification using any of the illustrativetechniques described herein. In order to determine publication status,for example, network entity 620 may search among content catalogs 691,692, 693, and 694 stored in memory of respective network entities 621,622, 623, and 624. Network entity 620 may generate digital signatures ofimage 670, or objects depicted therein, for comparison with contentstored in content catalogs 691, 692, 693, and 694. Network entities621-624 may store respective content catalogs 691-694 (e.g., collectionsof content, or a gallery), associated metadata, along with referencesignatures of images stored therein, reference signatures of objectsdepicted therein, or a combination thereof. Network entity 620 maycompare digital signatures associated with image 570 with referencedigital signatures of images of content catalogs 691-694. If a match isfound, or otherwise network entity 620 determines that image 570corresponds to an image of content catalogs 691-694, and the matchingimage is published, then network entity 620 may determine that image 670is published. If a match is found, or otherwise network entity 620determines that image 570 corresponds to an image of content catalogs691-694, and the matching image is private, then network entity 620 maydetermine that image 670 is not published. If no match is found, orotherwise network entity 620 determines that image 570 does notcorrespond to any image of content catalogs 691-694 then network entity620 may determine that image 670 is not published, publishable, notpublishable, or determine another publication status based onpredetermined criteria. To illustrate, network 620 may determine thatimage 695 of content catalog 694 of network entity 624 matches image670, and accordingly may access metadata of image 695 to determinewhether image 695 is public, or otherwise suitable for contentmodification.

In an illustrative example, network entity 620 may receive a request forcontent modification, a request for content suitable for contentmodification, one or more criteria for analyzing content, or acombination thereof. In response, network entity 620 may search forcontent stored in memory of network entity 620, among network entities621-624, or a combination thereof. In some circumstances, network entity620 may receive one or more digital signatures of an image, orattributes depicted therein, and compare the one or more digitalsignatures with reference digital signatures (e.g., stored in any ofnetwork entities 620-624). In some circumstances, network entity 620 mayreceive a request for content for which content modification issuitable. In response, network entity 620 may search among, and analyze,content of network entities 620-624). In some circumstances, networkentity 620 may receive a request for content for which contentmodification of a particular type is suitable. In response, networkentity 620 may search among, and analyze, content of network entities620-624). In some circumstances, network entity 620 may receive one ormore criteria for content modification suitability analysis. Inresponse, network entity 620 may search among, and analyze, content ofnetwork entities 620-624), and analyze the content based on the one ormore criteria (e.g., generating or updating metadata after analysis).

In an illustrative example, a photo selection engine may operate oncontrol circuitry of network entity 620, identifying and retrievingcontent from one or more of network entities 621-624. Network entities620-624, or a combination thereof, may host a tagging engine, whichgenerates tags as content is created and uploaded (e.g., for content inany of content catalogs 691-694). A content modification engine mayoperate on control circuitry of any or all of network entities 620-624,analyzing content for content modification suitability.

FIG. 7 shows an illustrative display of user device 700 with anapplication operating, in accordance with some embodiments of thepresent disclosure. User device may be the same as, similar to, ordifferent than any of user device 450 of FIG. 4, user devices 550-553 ofFIG. 5, for example. In some embodiments, user device 700 is asmartphone. The display of user device 700 shows an application, which,as illustrated, may include a social media feed or other platform fordisplaying and sharing content. The application includes toolbar 701,search field 702, image gallery 703, input region 704, and region 705.Although shown as separate regions, application includes toolbar 701,search field 702, image gallery 703, input region 704, and region 705each may be omitted, combined with other regions, further partitioned,or otherwise modified from the illustration of FIG. 7, in accordancewith some embodiments of the present disclosure.

Toolbar 701 includes soft buttons or selectable icons that provideindications or commands to the application from the user (e.g., viahaptic interaction with a touchscreen or input to a displayed keyboard).In some embodiments, toolbar 701 provides options for the user tointeract with the application. For example, toolbar 701 may be used forchanging the display, launching add-ins or applications (e.g., cameraapplications, messaging applications), searching (e.g., among contacts,content, functions, memory storage), any other suitable function, or anycombination thereof.

Search field 702 includes a region for text input by a user (e.g., usinga displayed keyboard), which may be used to generate a search query. Insome embodiments, the user may input text to search for content,contacts, or other information.

Image gallery 703 includes thumbnails or other representations ofcontent stored in memory of user device 700, stored in memory of anetwork entity hosting the application, any other suitable content, orany combination thereof. As illustrated, for example, three imagethumbnails are shown, and the larger catalog may be perused byscrolling. Image gallery 703, for example, may correspond to any of thecontent catalogs of FIG. 6, wherein the respective network entityincludes user device 700. In some embodiments, some, all, or none of theimages in image gallery 703 include advertising. For example, images ofimage gallery 703 may be images as created, as stored, as modified, asuploaded, as downloaded, or otherwise in any suitable state based onhistory of the content. In some embodiments, the user may select one ormore content items of image gallery 703 for further processing, analysisfor modification, downloading, uploading, sharing, any other suitableaction, or any combination thereof.

Input region 704 may include a region for text input by the user,selectable icons or soft buttons, fields, displayed user profileinformation, any other suitable displayed objects, or any combinationthereof. In some embodiments, input region 704 includes a representationof a file directory or file folder contents, allowing the user to selector browse stored content. In some embodiments, input region 704 includesfields for user input for analysis performed in region 705.

Region 705 includes a display a representation of content undergoinganalysis for content modification, details of the analysis (e.g.,progress, output, current determination, completed determinations,metadata), regions for user input, any other suitable features, or anycombination thereof. In some embodiments, content selected by the useris analyzed, and a representation of the analysis is presented in region705.

In an illustrative example, the user may launch the application on userdevice 700, which provides a display of image gallery 703 based onimages stored in user device 700. The user selects an image of imagegallery 703, and enters text information to input region 704, which theapplication writes to metadata associated with the selected image. Anymodifications, such filtering, cropping, or other photo edits to theselected image are shown in region 705. Accordingly, the user may usethe application to manage storing an image, modifying an image,generating metadata for an image, or otherwise interacting with animage.

In an illustrative example, the user may launch the application on userdevice 700, which provides a display of image gallery 703 based onimages stored in user device 700. The application may receive a requestfor content modification, and analyze images of image gallery 703 todetermine an image suitable for content modification. The applicationmay then modify the image with content modification and display themodified image in region 705 (e.g., as an advertisement in the user'sfeed displayed to the user).

In an illustrative example, the user may launch the application on userdevice 700, which searches for, accesses, or otherwise identifies aplurality of images stored in memory of a plurality of network entities,optionally including one or more user devices. The application mayreceive a request for content modification from the user, and analyzethe plurality of images to determine an image suitable for contentmodification. The application may then select and modify the image. Themodified image may be stored in memory of, or displayed on a display of,any suitable network entity, including user device 700.

In an illustrative example, the application may operate on user device700, and the application may receive a considered image for analysis.The application may search for, accesses, or otherwise identify aplurality of images stored in memory of a plurality of network entities,optionally including one or more user devices. In some embodiments, theapplication may determine if the considered image is published bycomparing one or more digital signatures of the considered image withreference signatures of the plurality of images. In some embodiments,the application may determine if the considered image is published bycomparing metadata associated with the considered image with metadataassociated with the plurality of images.

A user, or network entity may access content, an application, and otherfeatures from one or more devices, one or more network-connecteddevices, one or more electronic devices having a display, or acombination thereof, for example. Any of the illustrative techniques ofthe present disclosure may be implemented by a user device, anetwork-connected device, a device providing a display to a user, or anyother suitable control circuitry configured to identify and analyzecontent. FIG. 8 shows generalized embodiments of an illustrativenetwork-connected device. User device 800, as illustrated, includessmart phone configured to generate content, display content, modifycontent, or otherwise provide for an interaction between the user thecontent. Computing system 801 may include a server or other computingdevice that is connected to a communications network, and optionally mayinclude, or be communicatively coupled to, display 812, audio equipment814, and input interface 810. In some embodiments, display 812 mayinclude a television display or a computer display. In some embodiments,user interface input 810 is a remote-control device. Network device 816may include one or more circuit boards. In some embodiments, the one ormore circuit boards include processing circuitry, control circuitry, andstorage (e.g., RAM, ROM, Hard Disk, Removable Disk, etc.). In someembodiments, circuit boards include an input/output path. Each one ofuser equipment device 800 and computing system 801 may receive contentand data via input/output (hereinafter “I/O”) path 802. I/O path 802 mayprovide content and data to control circuitry 804, which includesprocessing circuitry 806 and storage 808. Control circuitry 804 may beused to send and receive commands, requests, and other suitable datausing I/O path 802. I/O path 802 may connect control circuitry 804 (andspecifically processing circuitry 806) to one or more communicationspaths (described below). I/O functions may be provided by one or more ofthese communications paths but are shown as a single path in FIG. 8 toavoid overcomplicating the drawing. While network device 816 is shown inFIG. 8 for illustration, any suitable computing device having processingcircuitry, control circuitry, and storage may be used in accordance withthe present disclosure. For example, network device 816 may include, bereplaced by, or complemented by, a personal computer (e.g., a notebook,a laptop, a desktop), a network-based server hosting a user-accessibleclient device, a non-user-owned device, any other suitable device, orany combination thereof.

Control circuitry 804 may be based on any suitable processing circuitrysuch as processing circuitry 806. As referred to herein, processingcircuitry should be understood to mean circuitry based on one or moremicroprocessors, microcontrollers, digital signal processors,programmable logic devices, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), etc., and may includea multi-core processor (e.g., dual-core, quad-core, hexa-core, or anysuitable number of cores) or supercomputer. In some embodiments,processing circuitry is distributed across multiple separate processorsor processing units, for example, multiple of the same type ofprocessing units (e.g., two Intel Core i7 processors) or multipledifferent processors (e.g., an Intel Core i5 processor and an Intel Corei7 processor). In some embodiments, control circuitry 804 executesinstructions for an application stored in memory (e.g., storage 808).Specifically, control circuitry 804 may be instructed by the applicationto perform the functions discussed above and below. For example, theapplication may provide instructions to control circuitry 804 togenerate the media guidance displays. In some implementations, anyaction performed by control circuitry 804 may be based on instructionsreceived from the application.

In some client-server-based embodiments, control circuitry 804 includescommunications circuitry suitable for communicating with an applicationserver or other networks or servers. The instructions for carrying outthe above-mentioned functionality may be stored on the applicationserver. Communications circuitry may include a cable modem, anintegrated-services digital network (ISDN) modem, a digital subscriberline (DSL) modem, a telephone modem, an ethernet card, or a wirelessmodem for communications with other equipment, or any other suitablecommunications circuitry. Such communications may involve the Internetor any other suitable communications networks or paths. In addition,communications circuitry may include circuitry that enables peer-to-peercommunication of user equipment devices, or communication of userequipment devices in locations remote from each other (described in moredetail below).

Memory may be an electronic storage device such as storage 808 that ispart of control circuitry 804. As referred to herein, the phrase“electronic storage device” or “storage device” should be understood tomean any device for storing electronic data, computer software, orfirmware, such as random-access memory, read-only memory, hard drives,optical drives, solid state devices, quantum storage devices, gamingconsoles, gaming media, or any other suitable fixed or removable storagedevices, and/or any combination of the same. Storage 808 may be used tostore various types of content described herein as well as mediaguidance data described above. Nonvolatile memory may also be used(e.g., to launch a boot-up routine and other instructions). Cloud-basedstorage, for example, may be used to supplement storage 808 or insteadof storage 808.

A user may send instructions to control circuitry 804 using user inputinterface 310. Input interface 810, display 812, or both, may include atouchscreen configured to provide a display and receive haptic input.For example, the touchscreen may be configured to receive haptic inputfrom a finger, a stylus, or both. In some embodiments, user device 800may include a front-facing screen and a rear-facing screen, multiplefront screens, or multiple angled screens. In some embodiments, inputinterface 810 includes a remote-control device having one or moremicrophones, buttons, keypads, any other components configured toreceive user input or combinations thereof. For example, input interface810 may include a handheld remote-control device having an alphanumerickeypad and option buttons. In a further example, input interface 810 mayinclude a handheld remote-control device having a microphone and controlcircuitry configured to receive and identify voice commands and transmitinformation to network device 816. In some embodiments, a user mayaccess computing system 801 using a shell such as a secure shell (SSH),an unsecured shell, a virtual private network, a login (e.g., hosted byan application), any other suitable technique for accessing computingsystem 801, or any combination thereof.

Audio equipment 814 may be provided as integrated with other elements ofeach one of user device 800 (e.g., and optionally of computing system801, although not shown) or may be stand-alone units. The audiocomponent of videos and other content displayed on display 812 may beplayed through speakers of audio equipment 814. In some embodiments, theaudio may be distributed to a receiver (not shown), which processes andoutputs the audio via speakers of audio equipment 814. In someembodiments, for example, control circuitry 804 is configured to provideaudio cues to a user, or other audio feedback to a user, using speakersof audio equipment 814. Audio equipment 814 may include a microphoneconfigured to receive audio input such as voice commands or speech. Forexample, a user may speak letters or words that are received by themicrophone and converted to text by control circuitry 804. In a furtherexample, a user may voice commands that are received by the microphoneand recognized by control circuitry 804.

An application (e.g., for generating a display) may be implemented usingany suitable architecture. For example, a stand-alone application may bewholly implemented on each one of user device 800 and computing system801. In some such embodiments, instructions of the application arestored locally (e.g., in storage 808), and data for use by theapplication is downloaded on a periodic basis (e.g., from an out-of-bandfeed, from an Internet resource, or using another suitable approach).Control circuitry 804 may retrieve instructions of the application fromstorage 808 and process the instructions to generate any of the displaysdiscussed herein. Based on the processed instructions, control circuitry804 may determine what action to perform when input is received frominput interface 810. For example, movement of a cursor on a displayup/down may be indicated by the processed instructions when inputinterface 810 indicates that an up/down button was selected. Anapplication and/or any instructions for performing any of theembodiments discussed herein may be encoded on computer-readable media.Computer-readable media includes any media capable of storing data. Thecomputer-readable media may be transitory, including, but not limitedto, propagating electrical or electromagnetic signals, or may benon-transitory including, but not limited to, volatile and non-volatilecomputer memory or storage devices such as a hard disk, floppy disk, USBdrive, DVD, CD, media card, register memory, processor cache, RandomAccess Memory (RAM), etc.

In some embodiments, the application is a client-server-basedapplication. Data for use by a thick or thin client implemented on eachone of user device 800 and computing system 801 is retrieved on-demandby issuing requests to a server remote from each one of user device 800and computing system 801. For example, the remote server may store theinstructions for the application in a storage device. The remote servermay process the stored instructions using circuitry (e.g., controlcircuitry 804) and generate the displays discussed above and below. Theclient device may receive the displays generated by the remote serverand may display the content of the displays locally on user device 800.This way, the processing of the instructions is performed remotely bythe server while the resulting displays (e.g., that may include text, akeyboard, or other visuals) are provided locally on user device 800.User device 800 may receive inputs from the user via input interface 810and transmit those inputs to the remote server for processing andgenerating the corresponding displays. For example, user device 800 maytransmit a communication to the remote server indicating that an up/downbutton was selected via input interface 810. The remote server mayprocess instructions in accordance with that input and generate adisplay of the application corresponding to the input (e.g., a displaythat moves a cursor up/down). The generated display is then transmittedto user device 800 for presentation to the user.

In some embodiments, the application is downloaded and interpreted orotherwise run by an interpreter or virtual machine (e.g., run by controlcircuitry 804). In some embodiments, the application may be encoded inthe ETV Binary Interchange Format (EBIF), received by control circuitry804 as part of a suitable feed, and interpreted by a user agent runningon control circuitry 804. For example, the application may be an EBIFapplication. In some embodiments, the application may be defined by aseries of JAVA-based files that are received and run by a local virtualmachine or other suitable middleware executed by control circuitry 804.

In some embodiments, a system may include a user interface, withprocessing occurring remotely (e.g., at a network entity), asillustrated in FIGS. 4-6, for example.

FIG. 9 is a flowchart of an illustrative process 900 for analyzingcontent for content modification, in accordance with some embodiments ofthe present disclosure. Process 900 may be implemented by any suitabledevice or combination of devices including, any of the devicesillustrated in FIGS. 4-8.

Step 902 includes control circuitry identifying content stored in memoryof one or more devices. In some embodiments, the control circuitryidentifies the content based on receiving a request to analyze thecontent. In some embodiments, the control circuitry identifies thecontent during a search process wherein a plurality of content isconsidered. The control circuitry and memory may, but need not, residewithin the same device.

Step 904 includes control circuitry extracting one or more objectsdepicted in the content. In some embodiments, the control circuitryutilizes one or more known techniques for object recognition including,for example, edge detection, color pattern recognition, partial linearfiltering, regression algorithms, neural network pattern recognition, ora combination thereof. In some embodiments, the extracting the object isbased on metadata associated with the content. For example, the metadatamay include one or more tags that indicate that the content includes aperson, a place, an item, any other suitable object, or a combinationthereof.

Step 906 includes control circuitry generating signatures for eachobject extracted at step 904. Signatures include, for example,information about one or more pixels associated with an object of thecontent, information about the object (e.g., shape information, sizeinformation, feature information), any other suitable informationdescribing an object, or any combination thereof.

Step 908 includes control circuitry comparing the signatures toreference signatures. In some embodiments, the control circuitryutilizes one or more known techniques for comparing digital signaturessuch as mean squared error, structural similarity, or other suitabletechnique. In some circumstances, the control circuitry determines amatch between the signatures and reference signatures based on aprevious comparison of the determined object to known objects, previousdetermination of reference signatures of objects, or other previousanalysis.

Step 910 includes control circuitry determining context informationabout the content. In some embodiments, the control circuitry determinescontext information about the content based on the comparison of step908. Context information includes, for example, information about whatis depicted in the content including objects, entities, events, scenes,places, persons, items, or other aspects depicted in the content.

Step 912 includes control circuitry determining a publication status ofthe content. In some embodiments, the control circuitry determineswhether content is public, publicly available, private, shared, orundesignated. In some embodiments, publication identifier 251 accessesmetadata to determine whether content has been previously tagged aspublic or private. In some embodiments, the control circuitry determineswhether the content is public based on whether is publicly available.For example, the control circuitry may generate one or more digitalsignatures corresponding to the content that describe features andaspects of the content. The control circuitry then compares the digitalsignatures against one or more reference signatures of publiclyavailable content to identify the content as being publicly available.

Step 914 includes control circuitry generating a content modificationflag. The content modification flag may include one or more statusidentifiers (e.g., “restricted,” “unrestricted,” “conditionallyrestricted,”), flag values (e.g., binary values, a score or ranking, orother numerical value), or a combination thereof. In some embodiments,the control circuitry generates a score or other ranking value, whichprovides a relative indication of how suitable content is for contentmodification (e.g., for comparison with other content for the mostsuitable content for a particular instance of content modification). Insome embodiments, the control circuitry generates a binary contentmodification flag, indicating whether the content is suitable formodification or not. In some embodiments, the control circuitrygenerates a list or collection of one or more keywords or designations,indicating for which products the content is suitable for modification,what type of modification is suitable (e.g., overlay, logo replacement,logo addition, logo removal, etc.). A content modification flag mayinclude any suitable information arranged in any suitable format, inaccordance with the present disclosure.

Step 916 includes control circuitry storing the content modificationflag in memory storage of a suitable device. After generating thecontent modification flat at step 914, the control circuitry may thenupdate metadata, generate metadata, or both. For example, the controlcircuitry may add tags to existing metadata. In some embodiments, thecontrol circuitry reads and writes files to memory (e.g., local memorystorage or remote storage via a communications network). For example, insome embodiments, the control circuitry opens a content data file andmetadata, generates or updates tags or otherwise changes the metadata,writes this information to an appropriate file and saves the files insuitable storage. In some embodiments, the control circuitry updates oneor more instances of metadata with the content modification flags. Forexample, a plurality of network entities may store instances of thecontent and metadata, and the control circuitry may update all, some, orat least one of the instances of metadata.

FIG. 10 is a flowchart of illustrative process 1000 for creating andmodifying content, in accordance with some embodiments of the presentdisclosure. As illustrated, process 1000 includes process 1001 forcreating content and associated metadata, and processes 1011 foranalyzing and modifying the content. Process 1000 may be implemented byany suitable device or combination of devices including, any of thedevices illustrated in FIGS. 4-8.

Step 1002 includes control circuitry generating content. In someembodiments, the control circuitry generates content based on a cameraor imaging application implemented on a user device. For example, a usermay access the application, and content may be generated based on inputfrom the user (e.g., capturing a photo by the user pressing a softbutton). In a further example, the application may select existingcontent and modify, convert, copy, or other generate new content basedon the existing content (e.g., with or without user input). Generatingcontent may include, for example, storing one or more data files,storing one or more metadata files,

Step 1004 includes control circuitry receiving and/or generating contentinformation. In some circumstances, content is generated at step 1002and stored in memory of a user device. At some later time, for example,the content may be identified and received from memory by the controlcircuitry. In a further example, content is generated at step 1002, andcontent information (e.g., metadata) is received, retrieved, generatedor a combination thereof. In some embodiments, steps 1002 and 1004 arecombined, with content data and metadata being generated atapproximately the same time.

Step 1006 includes control circuitry storing content, associatedmetadata or both. In some embodiments, the control circuitry reads andwrites files to memory (e.g., local memory storage or remote storage viaa communications network). For example, in some embodiments, the controlcircuitry generates a content data file and metadata file for storage.In a further example, the control circuitry writes this information toan appropriate existing file and saves the files in suitable storage.

In an illustrative example, steps 1002, 1004, and 1006 may be performedat substantially the same time, such as when an image is created andstored. Process 1001 may be performed by a user device based on inputfrom a user, by a network entity based on an application, or acombination thereof. In some embodiments, any of steps 1002, 1004, and1006 may be omitted combined with other steps, or otherwise modified, inaccordance with the present disclosure.

Step 1012 includes control circuitry retrieving content, metadata, orboth. For example, the control circuitry may retrieve the content andassociated metadata stored during process 1001, or any other suitablecontent and associated metadata. The content and metadata, which mayinclude one or more datafiles, may be stored in any suitable memorystorage (e.g., storage 808). In some embodiments, the control circuitryretrieves the content and metadata in response to identifying thecontent, which is stored in memory of one or more devices. For example,the control circuitry may identify the content based on receiving arequest to analyze the content. In some embodiments, the controlcircuitry identifies and/or retrieves the content during a searchprocess wherein a plurality of content is considered. The controlcircuitry and memory wherein the content is stored may, but need not,reside within the same device.

Step 1014 includes control circuitry receiving a request for contentmodification. Step 1014 may be performed before step 1012 (e.g., contentis retrieved in response to the request for content modification), afterstep 1012 (e.g., content is retrieved, and a request is subsequentlyreceived), or at the same time as step 1012 (e.g., a request isreceived, wherein the request includes identified content for analysis).In some embodiments, step 1014 includes two requests: a first requestfor analysis of content, and a second request for content modification.

Step 1016 includes control circuitry analyzing whether the content ispermissioned for modification. Step 1016 includes any of the analysesdescribed in the context of FIGS. 1-3, for example. The controlcircuitry performs any or all of publication analysis (e.g., usingpublication identifier 251), restriction analysis (e.g., usingrestriction identifier 252), context analysis (e.g., using contextidentifier 253), compatibility analysis (e.g., using compatibilityidentifier 254), and classification analysis (e.g., using imageclassifier 255), along with any other suitable analyses. In someembodiments, the content is analyzed without respect to a particularcontent modification (e.g., whether the content is permissioned for anycontent modification). In some embodiments, the content is analyzed withrespect to a particular modification (e.g., whether the content ispermissioned for the particular modification).

Step 1018 includes control circuitry determining whether the content ispermissioned for modification based on the analysis. In someembodiments, the control circuitry determines whether the content ispermissioned based on a content modification flag or other indicator inmetadata (e.g., generated during step 1016). In some embodiments, steps1016 and 1018 are combined or otherwise performed at the same time. Thecontrol circuitry may determine that content is permissioned,non-permissioned, or conditionally permissioned

In some embodiments, a binary metric is used to determine whethercontent is permissioned at step 1018. For example, a contentmodification flag of “0” or “1,” or other binary indicator, may bedetermined based the analyses of step 1016. If the flag corresponds to apermissioned value, then the content is determined to be permissioned,and if the flag corresponds to a non-permissioned value, then thecontent is determined to be not permissioned.

In some embodiments, a score or other non-binary metric is determinedand compared to a threshold to determine whether content is permissionedat step 1018. For example, a permission index may be determined basedthe analyses of step 1016, and the permission index may be compared witha threshold. If the permission index exceeds the threshold, the contentis determined to be permissioned, and if the permission index does notexceed the threshold, the content is determined to be not permissioned.

Step 1020 includes control circuitry modifying the content, themetadata, or both. The control circuitry may, for example, modify theanalyzed and permissioned content with content modification, updatemetadata associated with the analyzed and permissioned content with anindicator, or update metadata associated with the analyzed andnon-permissioned content with an indicator.

In an illustrative example, the control circuitry may receive a requestfor content modification at step 1014, and then identify and retrieve aplurality of candidate content at step 1012. The control circuitry maythen analyze whether the candidate content is permissioned for thecontent modification at step 1016, and if determined to be permissionedat step 1018, the control circuitry may proceed to modify the content atstep 1020.

In an illustrative example, the control circuitry may receive a requestfor content modification at step 1014, which includes candidate contentand metadata. The control circuitry may then analyze whether thecandidate content is permissioned for the content modification at step1016, and if determined to be permissioned at step 1018, the controlcircuitry may proceed to modify the metadata with a content modificationflag at step 1020.

The illustrative steps and processes of FIGS. 9-10 may be implantedusing any suitable systems, devices, and arrangements described in thecontext of FIGS. 1-8. For example, processes 900, 1001, and 1011, or anysteps thereof, may be performed by control circuitry of a user device, anetwork entity, or both, based on content stored in memory of a userdevice, a network entity, or both. In a further example, the controlcircuitry may implement computer readable instructions of anapplication. In some embodiments, a tagging engine generates tags forcontent, a photo selection engine identifies, retrieves and parsescontent, and a content modification engine determines whether content issuitable for content modification (e.g., and may perform the contentmodification). The tagging engine may include any suitable aspects of,for example, content analyzer 250 and metadata manager 270. The photoselection analyzer may include any suitable aspects of, for example,content analyzer 250. The content modification engine may include anysuitable aspects of, for example, content analyzer 250.

In an illustrative example, process 1001 may be performed at a userdevice. For example, users' photos are typically stored in variouslocations including a user's personal devices (e.g., a smart phone orlaptop) and in the cloud with services such as Google Photos, Twitter,or Facebook (e.g., applications for managing content). In someembodiments, a user submits a photo to an application or service (e.g.,taking a photo on an iOS-based device, Android-based device or otheruser device and having it appear in a feed, or posting a photo in asocial media feed such as Facebook, Twitter, or similar services), orotherwise shares the photo (e.g., by emailing the photo or otherwiseincluding it as part of a message). A tagging engine (e.g., contentanalyzer 250 with metadata manager 270) tags the local copy of the photowith a unique indicator (e.g., as a suitable hash, flag or score),identifying the photo as having been published and, optionally,identifying the user or corresponding device. To illustrate, the taggingengine may determine that content emailed to a contact might still beconsidered private rather than published, while content submitted to acontest or website may be considered as published and no longer private.

In an illustrative example, a content modification tag may be generated(e.g., by content analyzer 250 with metadata manager 270) and stored instorage of a user device (e.g., user device 800). The tag may be storedin metadata of the content file, or as a separate file in a directory orlibrary structure of the user device. Storing the tag as metadata allowsany system to which the photo and metadata are subsequently copied toeasily determine that the photo has been previously published. In someembodiments, for example, the tagging engine identifies the photo andsends it to a photo selection engine (e.g., described below).

In an illustrative example, the tagging engine may be a plug-in to theapplication with which the photo is published. Alternatively, oradditionally, the tagging engine may be a feature of the operatingsystem of the user's device (e.g., user device 800). For example,applications use an operating system (OS) application programminginterface (API) call for discovering locally-stored photos and selectingthem for uploading, sharing, modifying, or other actions. In someembodiments, the called routine will tag (e.g., update metadata for) thephoto, for example, just before uploading the photo to a service orapplication. In some embodiments, if the tagging engine is resident onthe user's device, the tag is supplied as metadata with the photo to thereceiving service or application (e.g., Google Photos, Facebook,Twitter, etc.). Tags may be generated based on user input (e.g.,user-specified tags or “user-tagging”), based on analysis (e.g., ofcontent analyzer 250 as an “auto-tagging” process), or a combinationthereof. In some embodiments, the tagging engine may update metadatabased on user-tagging, auto-tagging, or both. For example, the taggingengine may weigh tags, with machine tags having a lower weight than auser tag.

In an illustrative example, the receiving service or applicationperforms the tagging (e.g., rather than the user device). Such servicesor applications may tag the photo upon upload (e.g., when includedinitially in a tweet or Facebook post), at the time the photo isactually shared (e.g., as opposed to when it is merely stored in afolder accessible only to the user), or at some other time (e.g., aspart of a scheduled analysis of content). The tag may be stored asmetadata associated with the photo or in a directory or librarystructure storing the photo, which may reside in any suitable memory. Insome embodiments, the photo is also tagged with any privacy settingsunder which the photo was published. For example, the photo may beconsidered published for some other users, for some restricted purposes,or other conditional publishing. In some embodiments, services andapplications may omit tagging altogether and provide an API by which aphoto selection engine may query the service for published photos.

A photo selection engine selects or confirms photos for contentmodification (e.g., by a suitable content modification engine). Thephoto selection engine may include any suitable aspects of contentanalyzer 250, for example.

In an illustrative example, the photo selection engine may be acomponent of a cloud service (e.g., executing on an application serverof Google Photo, Facebook, Twitter, or other such services), resident onthe user device (e.g., as part of a client of such services, or as partof the OS), or executed on a third-party server that is queried byapplications and services for photos appropriate for contentmodification (e.g., a “clearinghouse” model).

In an illustrative example, a content modification engine may query(e.g., provide a request to) the photo selection engine in a number ofways. In some embodiments, the content modification engine supplies auser identifier and, optionally, a number of attributes. The photoselection engine searches a dataset of photos (e.g., stored in memory ofone or more user devices or network entities) attempting to match theuser identifier and attributes. Matching photos are returned to thecontent modification engine.

In an illustrative example, the content modification engine supplies anactual photo (e.g., rather a user identifier and attributes). Thecontent modification engine (e.g., content analyzer 250 or modulesthereof) will determine whether the photo has a tag (as describedabove), or use pattern matching or other suitable processes (e.g., imageor object recognition) for determining whether there is a matching photoin its dataset. If there is a matching tag or photo, the photo selectionagent confirms via a suitable return value that the photo has beenpublished and is suitable for content modification.

In an illustrative example, the photo dataset of the photo selectionengine may take various forms. In some embodiments, the tagging engineidentifies photos and sends them to the photo selection engine. Thephoto selection engine stores and catalogs the photos, and accompanyingmetadata. In some embodiments, the photo selection engine mines a userdevice for tagged photos and catalogs the photos, optionally copying thephoto or cataloging its location. For example, the photo selectionengine may update or otherwise modify metadata associated with thephotos for later analysis (e.g., by the content modification engine).

In an illustrative example, the photo selection engine mines publicfeeds and web pages for content. In some embodiments, the photoselection engine uses contextual data or metadata to associate a photowith a user and track that the photo is published. In some embodiments,the photo selection engine uses pattern matching and/or imagerecognition techniques to identify who is in the picture. The photoselection engine may store a location of the photo, scrape the photo(e.g., from a content catalog based on metadata), request a copy of thephoto via an API of the publishing application or service (if offered),and catalog the photo and its attributes (e.g., by generating orupdating metadata associated with the photo). In some embodiments, thephoto selection engine builds the dataset by requesting published photosfrom applications or services (e.g., Google Photos, Facebook ortwitter).

In some embodiments the photo selection engine, before selecting orconfirming the photo for content modification, analyzes the publishedphoto and any metadata to determine whether the photo contains materialunsuitable for content modification (e.g., despite the fact the photohas been published). The photo selection engine uses pattern/imagerecognition to identify elements of the photo and determines whethersuch elements are on a blacklist (i.e., a collection of attributesrendering content restricted from modification). The photo's associatedmetadata may also be searched for blacklisted elements. If matches areidentified between the photo and the blacklist, the photo is notselected or confirmed for content modification. A blacklist isconstructed using any suitable technique, and may be unique to eachuser, general for all users, or specific to products considered forplacement. In some embodiments, the blacklist is pre-populated toinclude sensitive topics given certain cultural norms of the user. Forexample, the blacklist may be pre-populated to include images andkeywords associated with places of worship, funeral homes, or othervenues or objects generally regarded as inappropriate for contentmodification. In some embodiments, the blacklist is constructed overtime based on feedback from the user. For example, the photo selectionengine may provide an API by which applications (e.g., contentmodification engines or tagging engines) can provide feedback. In anillustrative example, a social media feed may offer a user interface(UI) element to allow a user to express displeasure with contentmodification in a given photo. In response to the user's use of theelement, the social media feed may communicate (e.g., generate a call)to the photo selection engine to register the user's displeasure (e.g.,using any suitable message, information, or tag). Arguments for thecommunication may include, for example, the photo itself, attributes ofthe photo, photo metadata, an identification of what was placed, whatwas replaced, privacy settings of the photo, any other information orimages suitable for including in the blacklist, or any combinationthereof. In an illustrative example, the photo selection engine mayapply facial recognition software to filter out content includingchildren for ethical purposes (e.g., to prevent using someone's child'slikeness to advertise products), legal reasons (e.g., data privacy lawsfor people under 18 years of age, or under 13 years of age),user-specified criteria (e.g., a user prefers not to have family photosconsidered for content modification), or a combination thereof. In someembodiments, the photo selection engine may use privacy settings of aphoto when determining whether content is suitable or otherwisepermissioned for content modification. For example, the photo selectionengine may determine not to select photos that have something less thana publicly-shared privacy setting.

The above-described embodiments of the present disclosure are presentedfor purposes of illustration and not of limitation, and the presentdisclosure is limited only by the claims that follow. Furthermore, itshould be noted that the features and limitations described in any oneembodiment may be applied to any other embodiment herein, and flowchartsor examples relating to one embodiment may be combined with any otherembodiment in a suitable manner, done in different orders, or done inparallel. In addition, the systems and methods described herein may beperformed in real time. It should also be noted, the systems and/ormethods described above may be applied to, or used in accordance with,other systems and/or methods.

1. A method for screening content for modification, the methodcomprising: identifying content associated with an entity; extractingone or more objects of the content; generating for each of the one ormore objects a signature; comparing each signature to a database ofreference signatures; determining context information based on thecomparison of each signature; determining a publication status of thecontent; generating a content modification flag based on the contextinformation and based on the publication status; and storing the contentmodification flag in data storage, wherein the content modification flagis linked to the content.
 2. The method of claim 1, further comprising:receiving a request for candidate content suitable for contentmodification; and determining whether the content is suitable forcontent modification based on the content modification flag, wherein: ifit is determined that the content is not suitable for contentmodification, denying the request, and if it is determined that thecontent is suitable for content modification, accepting the request. 3.The method of claim 1, wherein determining the publication status of thecontent comprises: determining a content signature of the content;comparing the content signature with reference content signatures of aplurality of content; and based on the comparison, determining whetherthe content signature matches a reference content signature.
 4. Themethod of claim 1, further comprising retrieving metadata linked to thecontent, wherein storing the content modification flag in the datastorage comprises: updating the metadata with the content modificationflag; and storing the updated metadata in the data storage.
 5. Themethod of claim 4, wherein the metadata comprises publicationinformation, and wherein determining the publication status is based onthe publication information.
 6. The method of claim 1, whereingenerating the content modification flag comprises determining one ormore categories of suitable types of products for content modification.7. The method of claim 1, wherein generating the content modificationflag comprises determining one or more portions of the content that aresuitable for content modification.
 8. The method of claim 1, wherein thecontent comprises an image, and wherein the content modification flag isindicative of whether the image is suitable for content modification. 9.The method of claim 8, wherein extracting the one or more objects of thecontent comprises identifying at least one of the group comprising aperson, a place, and an item represented by the content.
 10. The methodof claim 1, further comprising generating metadata for storage in thedata storage, wherein the metadata comprises one or more of the groupcomprising signatures of the one or more objects of the content, thecontext information, and the content modification flag.
 11. A system forscreening content for modification, the system comprising: data storageconfigured to store data; and control circuitry coupled to the datastorage and configured to: identify content associated with an entity;extract one or more objects of the content; generate for each of the oneor more objects a signature; compare each signature to a database ofreference signatures; determine context information based on thecomparison of each signature; determine a publication status of thecontent; generate a content modification flag based on the contextinformation and based on the publication status; and store the contentmodification flag in the data storage, wherein the content modificationflag is linked to the content.
 12. The system of claim 11, wherein thecontrol circuitry is further configured to: receive a request forcandidate content suitable for content modification; and determinewhether the content is suitable for content modification based on thecontent modification flag, wherein: if it is determined that the contentis not suitable for content modification, deny the request, and if it isdetermined that the content is suitable for content modification, acceptthe request.
 13. The system of claim 11, wherein the control circuitryis further configured to: determine a content signature of the content;compare the content signature with reference content signatures of aplurality of content; and based on the comparison, determine whether thecontent signature matches a reference content signature.
 14. The systemof claim 11, wherein the control circuitry is further configured to:retrieve metadata linked to the content; update the metadata with thecontent modification flag; and store the updated metadata in the datastorage.
 15. The system of claim 14, wherein the metadata comprisespublication information, and wherein determining the publication statusis based on the publication information.
 16. The system of claim 11,wherein the control circuitry is further configured to determine one ormore categories of suitable types of products for content modification.17. The system of claim 11, wherein the control circuitry is furtherconfigured to determine one or more portions of the content that aresuitable for content modification.
 18. The system of claim 11, whereinthe content comprises an image, and wherein the content modificationflag is indicative of whether the image is suitable for contentmodification.
 19. The system of claim 18, wherein the control circuitryis further configured to identify at least one of the group comprising aperson, a place, and an item represented by the content.
 20. The systemof claim 11, wherein the control circuitry is further configured togenerate metadata for storage in the data storage, wherein the metadatacomprises one or more of the group comprising signatures of the one ormore objects of the content, the context information, and the contentmodification flag. 21-30. (canceled)